Bound Together: Signs and Features in Multimedia Content Representation

This paper introduces new extensions to the semiotic model that allow the model to account for image features that characterize an audio, visual or audio-visual object. The treatment in this paper emphasizes visual content description. The framework and the associated construct of image features characterizing the visual object “binding” to conceptual terms used to describe the visual object is described and illustrated in terms of UML diagrams. Visual object identification, location and media temporal segmentation approaches are outlined. The setting construct developed by Parkes [17] is then replaced with the introduction of the existent context as the minimal unit for temporal decomposition in content description. This construct forms the basis of the content modeling for the upper level representation scheme adopted in the Automating Video Annotation (AVA) moving picture content annotation prototype tool. The analysis is supported through a worked example developed on a content sequence from an episode of children’s television.